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A Computer Simulation Investigating the Applicability of 
the ftasch Mode 1 with Varying Item Discriminations 

In the classical modet of Item analysis, two princtple 
characteristics of an Item merit a t ten t i on-- these are, of 
course, the item difficulty and Item discrimination* In many 
situations, these indices seem to offer the test user important, 
non'redundant information about his test* Most champions of 
the classical mod el wou Id be careful to admonish the user to 
be sensitive to the {nterde pendency of his results and the 
subjects who have yielded them* 

In i960, however , Rasch ( Probabl istic Mode 1 s for Some 
Intelligence and Attainment Tests , cited in Lord, F* and Novick, 
H> , 1 96 8 ; ^ijejy and Dawis , J^97\f^ ^^'9^^ Panchapakasarv, 
1969) presented three models to explain misreadings, number 
of words read, and gene rat ach i evemen t ; each of these is a 
two parameter model , encorporating only the. ability of the 
person and the difficulty of the measurement to explain the 
observed data* The most impressive implication of the modejs 
Is that Item calibration and individual measurement are inde' 
pendent of both each other and the situation in which they take 
p I ace * 

The suggestion that an examinee's item score depends on 
only his ability and the difficulty of the item is an inherently 
pleasing one to many people* Without test artifacts like item 
discrimination to get in the way, the individual is pitted 
clearly against his criterion, and would thus, one might expect. 
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supply us with neatly interpretable data« Whet ha r the picture 
Is as clear as this has yet, of course, to be shown* 

The present research artificially generated the results 
of several hypothetical tests for which the effects of item 
discriminations varied* Fit to Rasch*s assumptions was pre- 
dicted on the fact that his third model may be understood as 
a two-parameter logistic function* With this bridge to more 
general mod els, then, the Rasch assumption of equal dlscrlmln- 
abM [ ty could ^e tested* 

The clearest demonstration of the relationship between the 
person and the It em is the It em characteristic curve* Here 
one theorizes the latent ability of the person (or class of 
peop le) plotted against the probability of getting a particular 
Item correct* At least since fiullford (1936), It has been 

"aTslume^d^ l£"Kat%' wrfhTri ^th^ ab 1' ri^ry raTiger— o^f^he^tes^tv^t-h^I^s—pro 

babiUty is best described by the norma? ogive function. How- 
ever, the assumption of normal Ity has been seen by some to 
be delimiting* By positing the logistic test model, 

\ +eX neatly avoid the restriction since 

Haley (cited In BIrnbaum, 1968), has shown that If ^ (x) Is 
the cumu latlve normal distribution function k (x) - * [ ( 1 * 7) x] I < 0, 0 1 
for all X* Within the context of test theory this mode 1 takes 
on a specific form credited to BIrnbaum, Pg(9) = [ P (9-ct) L where 
a ts the difficulty of the Item, P ts the Item discrimination, 
and 9 the examinee ability* The probability density function of 
th i s mode lis: 
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where Ug is 1 if the examinee responds correctly on Item g 
and Ug is 0 if he does not (BirnLaum, 1968). This, then, is 
the most genera 1 statemen t of the 1 og I st i c mode 1 encor porat I ng 
maximum information about the item and the examinee* 

Rasch (1966 a,b) has presented a model which can be seen 
as a s i mp t i f i ca t ion of this, one in which X=3(9-a) can be 
explain ed in terms ofeandaalone, the item dt^crtmination (3) 
having been assumed cons tan t across I tems (hence, here equal 
to one). The Implications of this lie in the fact that one 
can estimate 9 Independently of a and vice versa* As Wright 
and Panchap^kasSn tl$6sy h^avfe Indicated', h'owewr, th-e model' 
Impt les that : 

1. the model is un I d I men s I ona 1 ; 

2* there are no strong relationships among persons 
or Items other than those specified by the model 
so . that responses of persons to it ems are 
stochastically I ndependent given their parameters 
i n the model ; 

3* Items and persons do not differ substantially with 
respect to other response factors not represented 
In the model such as item discrimination, person 
sensitivity, guessing, or ind I f f erence - (p . 2) 

The author added that since few can write Items as a predeter^ 

mined discriminating level, it Is most feasible to discard 

"grossly dissimilar items (p* i})»" resulting in a set of 
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Items with "simiiar discrimination and minimal guessing"* If 
one were to have a decision rule for doing this, he would then 
be assured a fortiori of building a test in conformity with 
the dasch model * 

The present discussion describes one solution to the 
establishment of such a criterion. A Monte Carlo computer 
prog ram using the Rasch mod el was designed to input person and 
it em para meters, generate probabMities of success, simulate 
a test' ta king situation, produce the raw it em score matrix, and 
estimate the parameters of the Rasch it em characteristic curve* 
All four subsections may be used independently of each other, 
parameters can be read in or generated internally, and link- 
ups with other subsections are determined on 1 y by the tntent 
of the user* The subsection which esti mates the pa rameter s of 
t he~ Ra"sch~^i tem ctia rac t^ i^s t i c~ cFr^ve wTll "Tccep^r~a^ 
a raw item score matrix or a matrix of probabilities of success* 
In addition, the data- gene rating function foil ows Bi rnbaum * s ( 1 968) 
three pa rameter mode ] t and the data calibration foil ows Wright 
and Panchapakesan (1363); this allowed the present methodology: 
generatton of data using Birnbaum for simulation, and analysis 
of this data using Wright and Panchapakesan is calibration 
based on Rasch's mod el* Poor calibration would then suggest 
lack of robustness of the Rasch calibration to departures from 
homogeneity of item discrimination* 

The Simulation Program 

There are three general foci in the present FORTRAN 
s imu 1 at ion * The first read s it em difficulties, discriminations. 
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or person abiHties, or generates them act^ording to user specr** 
flcati^ons* Following this, the parameters are combined according 
to the Birnbaum formulation into a person x item matrix of 
probabilities. In the second (and actual simulation) phase, 
a series of random numbers is generated, each number being 
between 0 and 1; these numbers are compared with the probabili** 
ties generated in phase one and the "raw data" matrix is 
generated according to the rule: 

aj o 1 if P(af=l} > random number 
aj = 0 if P(a}=l} < random number 

The matrix of aj'^ could have been read in at this point 
instead of being generated. 

The third phase involves item calibration based on either 
the matrix of raw item scores or the person X item matrix of 
pToBabTri t iesT genera ted Tn fhe fTrst p'ti^se o'f the si mu I'a^t^i^on * 
dasch (1366 a, b) has shown that, assuming the double parameter 
model, total unweighted scores, that is ^aj^ for person (or 
score group) n are sufficient statistics for latent ability, 
which is estimated by the person (or score group) parameter 
Wright and Panchapakesan (1363) have elaborated Rasch's original 
I east squares approach; in addition, they have presented a 
maximum likelihood estimation which is more precise. 

Several points need to be made about this estimation* 
First, there is one and only ability level for any one score 
(or score group). Second, item calibration (that is, determining 
the alpha or item difficulties) generally precedes person 
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measurement (determining the thetas or person abilities)* 
n 

Third, if E a.. = n, that is, if any item is gotten correctly 
j=I 'J 

by all n people, it is useless for calibration. Similarly, 
^ k 

if E a. = k, i.e., alt Items were gotten correctly by person 
i -t 

J, that person's ability cannot be est I ma ted , and his responses 
contain no infer ma tion concerning the relative diffixulties 
of the i tems . 

The least squares and maximum likelihood methods are 
briefly treated here; the discussion follows Wright and Pan- 
chapakesan closely. The estimation of the Item difficulty 
is based upon the assumption that, within any score group, 
the probabM i ty of success on i tem i ' s approxi matel y the pro- 
portion of people within that score group who produced a 
correct response to that Item. With the estimate of the dif- 
ficulty scaled so that the mean difficulty equals zero ~the 
standard error of est! mate is derived from the variances of 
these probabilities using the assumption that the actual re- 
sponses to a given ttem within a given score group are brnoml*^ 
ally distributed. Estimation of person measurement is exactly 
paral lei to th is . 

The maximum likelihood estimates are necessary only for 
item calibration; the item estimates, generated first, can be 
used to calculate directly person abilities. Initially, the 
Implicit equations for item difficulties and person abilities 
are sol ved s Imu I taneous ly , u$ I ng an i terat i ve procedure . Once 
the It ems have been calibrated, the ability estimate for any 
exam i nee deper>ds upon no thing but his total raw score. More- 
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over, any set of calibrated items may be combined to form a 
new test, and a similar set of implicit equations may be solved 
iteratlvely to. determine the ability estimate corresponding 
to any possible raw score on the new test. Additionally, these 
estimates of theta and alpha are used to calculate the standard 
error of estimate of the estimated difficulty. 

For each item its good nes s-of- f i t to the Rasch model ts 
computed by forming a standard deviate 

V(a.,)'/^ 

where aj* Is the obtain ed it em score for person J on it em i, 

£(ajj) is the estimate of ajj based on it em difficulties (a|) 

1/2 

and person ability (6j), and V{ajj) is the standard devia- 
tion of the aj J . The squares of the standard deviates summed 
over people yield an approximate X with N - 1 degrees of 
freedom which can be used to test the fit of item i to the 
model . 

P r ocedu r e 

The central concern of the present research was the effect 
of item discriminations on fit to the Rasch model. !t was 
believed that a certain tolerance is a^llowed in the application 
of the theory. Exactly how much, of course, was not known. 
Degree of fit would be based on the degree to which item 
discriminations were the same, that Is, did not vary among 
themselves* This degree of fit was therefore operational ized 
as the variance of the item d i sc r 1 m i na t i on s « For the present 
s i mu iat i ons, va r i ances were assumed to be .05* *10, «15> *20, 
and .25^ One run was also made at = 0 to indicate 
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degree of accuracy of the item calibration* As there was 
also some question about the shape of the distribution of 
these values, this quality was also varied* Three forms 
were used , norma I, uniform, and posit! vely skewed . This 
latter form is thought to be the most reasonable for a well* 
construct ed test since discrimination values should never be 
negative; with a mean of one, the distribution would skew right. 
The actual shape was oper a t i ona I i zed as approximately a chi* 
square distribution with one degree of freedom* Since the 
Rasch procedure automatically scales the person and it em param** 
eters in such a way as to make the average item discrimination 
equal to one, this value was taken as the mean of all distribu* 
tions studied. There were thus sixteen simulation runs, one 
for the pure Rasch model and three at each degree of discrimin* 
ati'On vari-ab'i-l'i'ty'; aM- had means equa>V to one-. AM parameters 
except item discrimination were held constant* For each run 
a test length of 3O items was employed, with item difficulties 
randomly sampled from a normal distribution wtth mean 0 and 
standard deviation 1. The obtained random sample wh fch was 
used for all runs, had a mean of *I13 and a sample standard 
deviation of *9^0, the range being from -1*553 to 2*070* 

For each of the sixteen simulation runs, two calibrations 
were performed* First, the person X item matrix of probabilities 
was calibrated* This appr ox i mates the result of administering 
the test to an infinite sample and calibrating the data obtained 
in the conventional manner * The only difference between this 
calibration and calibration on an infinite sample lies !n the 
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fact that when the a priori probability matrix is calibrated 
directly, the pa rameter s may vary continuously, while In cali- 
brating actual data the ability estimates take on a set of 
discrete values, corresponding to each possible total raw score 
between 1 and k-1 items correct on a k"item test* 

The second calibration of each of the sixteen simulation 
runs was performed on a data matrix obtained by simulating 
an administration of the test to 75 persons and analyzing the 
obtained raw data matrix. The abilities of these persons were 
sampled at fixed intervals from ^ normal distribution with mean 
0 and variance 1*5^ The obtained sample had a mean of 0.00 
and a sample variance of 1.^75* While it is known that the 
best item calibration is done with a good deal of rep 1 i cab i 1 i ty 
within each score group, hence large N (Whitely and Dawis, l97^)f 
the compu ter t tme and cos t were p roh i^b i-t i^ve f or this. The n of 
75 was considered sufficient because (1) an additional call" 
bration was obtained on an "infinite" sample, and (2) the 
75 ^'persons" us^d were "centered on the test", i*e*, the test 
was of exactly the right difficulty for them, resulting in a 
very efficient administration with respect to amount of infor* 
ma t i on obta i ned . 

In the computer model i all s i mu 1 a t i ons a 1 1 ow the i tem 
difficulties and d i sc rmi na t i ons and the Person abilities either 
to be read in or generated inter nally* For the present reseach 
all three were generated randomly with the following characteris" 
tics. For all runs, the item difficulties were the same, having 
been random ly selected form the unit norma 1 distribution* The 
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person abilities were also normally distributed about a mean 
of Zero, except they had a variance of four* With these data 
fixed, sixteen simulation runs were attempted* The first 
used a standard default option built into the program and 
generated a unit vector of item discriminations; this, then, 
was the run where the precision of the item calibration routine 
could be tested since the input was purely Rasch^conforming. 
Each of the rema ining simulations, however , devi ated from the 
Rasch assumption of similar discriminations in two ways. Five 
of the runs had discriminations uniformly distributed with 
mean Bqual to one and variance equal respectively to .05f 
• 10, • 1 5 f * 20 , and *25 , each run showi ng increasingly st ronger 
deviation from Rasch's assumption. For each of these runs, 
discriminations were ssnpled at fixed intervals from the appro- 
priate uniform d"i s t r Tbu f i on * The h^xt five ^s'imul^ari'ons had 

discriminations norma lly distributed around a mean of one and 
var i ances respect j vely , 05 t * 1 0, • 1 5 t * 20 , and • 25 * Va ) ues 
were once more samp led at fixed intervals* 

The rema ining five analyses were based on the chi-square 
distribution with one degree of freedom* This distribution 
has a mean of one and a variance of two* Data points were 
selected in the following manner* Since thirty item discrimln** 
ations were needed, the chi-square PDF was broken into thirty 
equal areas; the mean of each area constituted the preliminary 
data point* These thirty pc*ints, with their mean of one and 
variance of two, were then converted to a data set having a 
mean of one and a variance of *25 using a linear transforma- 

12 
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tlon* This set of points was adjusted slightly to obtain the 
desired range of discriminations while holding the first two 
moments constant , and , finally, the obta I ned set was 11 near ly 
transfored to each of a set of points having a mean of one 
and the variances used above (*05, *I0, *l5f *20, *25)* 
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RESULTS 

The results Indicated that the Rasch calibration pro- 
cedure Is robust to departures from homogeneity of Item dls^^ 
crimination but that any tendency for this robustness to be 
lost does conform to Rasch's assumption* Table I presents 
what are believed to be the salient characteristics of the 
calibration upon which one might focus* Both Item fit and 
person fit are described for P matrix calibration and new 
data matrix calibration* The criteria of interest are the 
mean fit and the most extreme point of lack of fit* While 
they are both self-explanatory, It is felt that the letter 
deserves some explication* The ex t reme Instance of misfit 
may be misinterpreted unless It Is borne In mind that ^ \) it 
Is a single score and by Its nature an ext reme one and 2) for 
many of the simulations, they are bellved to be ou tilers. 
As a last point, the standard errors of estimate of person 
and Item parameters a/e rarely less than *1 logtt and can be 
qu I te a bit g reator * 

Several patterns were noted In the results, and, while 
there have been no statistical tests to confirm them, they 
have been included he re* 

First, as Is quite striking In Table If the average Item 
fit for both the P matrix and the raw data matrix was Identi- 
cally 2ero (including rounding errors)* In contrast, the 
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fit of persons was zero or negligible for the fits with zero 
beta variance or variance of .05 of the distribution were 
normal or skewed. 

Second, the poorest fit seemed to be for the uniform 
distribution, where both the average and maximum misfits 
were quite a bit larger than for either of the other distri'" 
bu t i ons . 

ThI rd , the pat terns acorss i ncreas i ng var i ab i 1 i ty are 
clearer in the theoretical (P matrix) calibration; as can be 
expected, the random error introduced in the simulation of 
test^taking clouded the issue. Here, both the uniform and 
normal distributions of betas showed the expected patterns 
the fit of item and people became worse as the variance of 
the betas increased. The pattern for the skewed distribution 
was not so clear. 

SUPPIEHENTARY ANALYSIS 

An argument for the use of item discrimination may still 
be made in terms of the ex traction of max! ma 1 information 
from the test. In order to assess the degree to which unwe 1 gh ted 
total score appr ox i mates the mathematically correct scoring 
In which each response is weighted by its item's discrimination, 
the ma thema t ically correct scoring was correl a ted with the 
unwe ighted number of it ems correct for each simulation. The 
minimum of these correlations across all sixteen simu^^lating 
runs was .8069« The magnitude of these correlations suggests 
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that whatever loss of information the use of unweighted raw 
scores might entail could be compensated for by a slight 
Increase in test lengths This conclusion! unfortunately, can* 
not be general ized to the case where a test Is of Inappropriate 
difficulty for the examinees. )n this case, a correlation 
may be indue ed between it em difficulty and it em discrimination, 
because Items at one end of the continuum of Item difficulties 
represented in the test will function better, and hence appear 
more discriminating, than items at the other end of the dif* 
f I cu 1 ty con t i nuum . 

CONCLUSIONS 

The present research suggests that the lack of an item 
discrimination parameter in the Rasch model does not result 
in poor calibration in the presence of varying item discrim- 
inations^ While the robustness of the model to other departures 
from assumptions rema ins to be investigated, such studies 
are also indicated for the norma) ogive model , more general 
logistic models, etc. Until such time as it Is shown to be 
either Inadequate or inferior to some other model, the use 
of the simplist mod el is to be recommended , If only^on the 
basis of ma t hema tical elegance and the sufficiency of total 
number of it ems correct as a statistic for subject ability. 

The substitution of equal item discriminations, rather 
than maximum it em discriminations as a goal !n Item writing, 
appears counter-intuitive to the test construction expert 
steeped in classical test theory* Wh 1 1 e It is t rue that 
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a highly discriminating Item is capable of providing more 

Information concerning the placement of ^n Individual on 

the continuum of some latent trait, the highly discriminating 

It em functions over a narrower range of abilities than a 

less discriminating Item* An Item with perfect discrimination 

would provide complete 1 nf o rma tlon about a single point on 

the ability continuum and no information about any other point* 

Therefore, for any given test, there will exist an optimal 

range of discrlmiration* If the test characteristic curve 

Is to rise steeply through a narrow range of abilities, more 

highly discriminating items will be desirable than if the 

test Is to function over a broad range of abilities* 

No guidelines can be provided Indicating a specific 
range of item discriminations which may be tolerated, first, 
because the mode 1 Is highly robust to differing discriminations 
and, second, because In the actual application of the model 
the true values of the discriminations are unknwon * I tem 
discriminations are estimated following calibration, by re* 
gressing probability of success of ability in the (linear) 
logistic matric* The wrose the fit of an Item, the larger 
the standard error of estimate of its discrimination may be* 
In the light of these considerations, the authors suggest 
Wright's (1969) approximate X2 statistics for the evaluation 
of fit* 
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TABLE I 

Degree of Misfit of Item Calibration For All 



Fit of Items Fit of Persons 









P Matrix 


Data Matrix* 


P Matrix 


Data Matrix* 


Run 




Dist. 


Mean 
Misfit 


Max i mum 
Misfit 


Mean 
Misfit 


Maximum 
Misfit 


Mean ' 
Misfit 


Max Imum 
Misfit 


Mean 
Misfit 


Maxi mum 
Misfit 


I 


.00 


- 


0 


- .1127 


0 


- .5113 


.OOOC 


. I 132 


.0600 


- I .283^1 


2 


.05 


u 


0 


-2.8300 


0 


-3.3306 


- . Ok06 


3 .0608 


- .0893 


I .7688 


3 


.10 


u 


0 


-3 .0390 


0 


-3. 3803 


-. 1203 


2.7^81 


-.1363 


2. 1622 




.15 


u 


0 


-3.1833 


0 


-3.5172 




-.I696 


3. 0961 


-.2i»36 


-2.6026 


5 


.20 


u 


0 


-3.29^2 


0 


-3.7293 


-.2058 


3.0978 


-.2277 


I ,h50€> 


6 


•25 


u 


0 


3.38^3 


0 


-if. 3912 


-.2358 


3.098I 


-.36U 


-2.5156 


7 


.05 


N 


0 


. U53 


0 


.6251 


-.0027. 


.1310 


-.0075 


. 600^1 


8 


. 10 


N 


0 


. 2203 


0 




- .00^(9 


.2628 


-.0772 


.7857 


9 


.15 


N 


0 


.2827 


0 


.7216 


-.0071 


.'3882 


-.0607 


.8206 


10 


.20 


N 


0 


.3387 


0 


1 .0069 


- .0092 


.505^ 


-.05^(3 


.81 U 



^computed by score group 



TABLE 1 (cont t nued ) 
Degree of Misfit of Item Calibration for All 

Fit of Items Fit of Persons 









P Matrix 


Data Matrlx<^ 


P Matrix 


Data Matrix^ 


Ru n 


e 


D i s t . 


M A a n 

n 6 a n 
Misfit 


M a V 1 m i 1 m 
n a A 1 mU m 

Misfit 


M A a n 

n 6 a n 
Misfit 


M a V 1 m i 1 m 
n a A 1 m Um 

Misfit 

L 


M A a n 

n 6 a n 
Misfit 


na X 1 mum 
Misfit 


Mean 
Misfit 


n a A 1 mU m 

Misfit 


1 7 


.05 


s 


0 


.301 1 


0 




- .0060 


.1016 


-.0560 


1 .0583 


13 


. 10 


S 


0 




0 


1 .1696 


-.0287 


.1 139 


-.0656 


1 .59U 


\h 


.15 


S 


0 




0 


.9101 


.0237 


.35^(2 


.05^3 


1 .7754 


15 


.20 


S 


0 


.3889 


0 


.6^(27 




.^(361 


.0336 


-3533 


16 


.25 


S 


0 


.6251 


0 


1 .0^(91 


- .02h€> 


.3^3^ 


-.0^(51 
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TABLE 2 



Correla 1 1 ons Between the Unwe t gh ted Total Score 
Approx t mat r ons and the Weighted Total 
Scores for AU Simulations 



u lat i on 




Distribution 




- 

I. 


.00. 




. 9BA7 


2 


.05 


I) 


.8123 


3 


.10 


U 


.8195 


k 


.15 


U 


.8069 


5 


.20 


U 


.85^7 


6 


.25 




.8183 


7 


.05 


N 


.9921 


8 


.10 


N 


.9875 


9 


.15 


N 


.9877 


10 


.20 


N 


.9885 


1 1 


.25 


N 


.9887 


12 


.05 


S 


.9888 


13 


.10 


S 


.9851 


14 


.15 


S 


.9827 


15 


.20 


S 


.9872 


16 


.25 


S 


.9786 
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Limttatfons of the Present Re search and 



Suggestions for Future Re search 



In this study, the only source of misfit which was in- 
troduced into the data was nonhomogene i ty of item discrimination* 
The calibration procedure proved quite robust to perturbations 
of this kind* Actual data, however , a re influenced by a wide 
variety of effects, e*g*, guessing, carelessness when it ems 
are too easy, practice effects which dis*rract the shape of 
the it em characteristic curve and/or induce violations of 
the assumption of local independence of persons and it ems* 

These additional sources of misfit raise several questions: 

1* )f more than one parameter is to be estimated 

for each it em, is discrimination the best choice 
to accompany difficulty, or wou Id more variance 
be accounted for by a pa ra meter representing, 
say, level of asymptote of the it em cha rac ter i s t i c 
curve (sensitivity to guessing)? 

2* Wou Id the Rasch calibration procedu re be less 

robust to variation in item discrimination if 

those variations occured in the context of other 
sources of mi sf i t ? 

3' If variations in item discrimination alone do not 
preclude the use of the Rasch model, what evidence 
is there that the normal ogive mode 1 is super t or 
to the Rasch model in fitting actual data?' 
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